Why we need an independent index of the Web

Search engine indexes function as a “local copy of the web”1, forming the foundation of every search engine. Search engines need to look for new documents constantly, detect changes made to existing documents, and remove documents from the index when they are no longer available on the web. When one considers that the web comprises many billions of documents which are constantly changing, the challenge search engines face becomes clear. It is impossible to maintain a perfectly complete and current index.2 The pool of data changes thousands of times each second. No search engine can keep up with this rapid pace of change.3 The “local copy of the web” can thus be viewed as the holy grail of web indexing at best — in practice, different search engines will always attain a varied degree of success in pursuing this goal.4